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ABSTRACT 


The shift towards qualifying performance and accoimting for results has 
dramatically changed the way Government executes public policy objectives. The advent 
of the Government Performance and Results Act (GPRA) places the responsibility for 
gathering this information upon each Federal activity subject to its provisions. The Army 
S mall Business Innovation Research (SBIR) program must now find a way to qualify its 
performance and determine what results are derived from a program that expends in 
excess of $100,000,000 annually on research. This thesis analyses Army SBIR 
commercialization rates against a National Science Foundation study of DoD Fast Track 
and DoD Control Group awards. It provides an objective measure of program results that 
program officials can use to submit their annual GPRA performance reports. The thesis 
studied 37 SBIR Phase II firms and established a performance baseline. The thesis 
concludes that Army SBIR awards are outperforming DoD Fast Track and DoD Control 
Groups in the critical area of average commercial sales per award. It recommends a 
reduced focus on outside investment and a survey strategy that uses small sample sizes to 
qualify program performance. It concludes with a proposed survey instrument that Army 
SBIR managers can use to capture future program outcomes. 


V 


THIS PAGE INTENTIONALLY LEFT BLANK 


VI 


TABLE OF CONTENTS 


1. INTRODUCTION.1 

A. GENERAL.1 

1. The Problem.1 

2. Potential Solutions.2 

3. What Happens if the Problem Remains Unanswered? ...........................2 

B. OBJECTIVES.2 

C. RESEARCH QUESTIONS.2 

1. Primary Research Question.3 

2. Subsidiary Research Questions.3 

D. SCOPE, LIMITATIONS, AND ASSUMPTIONS.3 

1. Scope.3 

2. Limitations.3 

3. Assumptions.4 

E. DEFINITIONS AND ABBREVIATIONS.4 

1. Research and Development.4 

2. Commercialization. 5 

3. Fast Track.5 

4. Extramural Budget.5 

5. Funding Agreement.5 

6. Subcontract.6 

7. Small Business Concern..6 

H. BACKGROUND.7 

A. GENERAL.7 

B. SBIR PROGRAM INTRODUCTION-7 

C. KEY PROVISIONS OF THE ACT_8 

D. SIGNIFICANT DEVELOPMENTS.8 

E. TOPIC GENERATION.:..9 

F. SBIR PROGRAM GOALS-i.10 

G. SBIR PROGRAM PHASES.11 

1. Phase 1.11 

2. Phase II.11 

3. Phase III.12 

H. ARMY RESEARCH OFFICE ORGANIZATIONAL STRUCTURE__12 

1. Organizational Structure.12 

2. Mission.12 

3. Proposal Evaluation.13 

4. SBIR Marketing Assistance.13 

I. FAST TRACK PROGRAM...13 

J. PRIOR STUDIES.14 

K GOVERNMENT PERFORMANCE AND RESULTS ACT.14 

L. SUMMARY.15 

vii 












































III. RESEARCH METHODOLOGY. 17 

A. GENERAL." 

B. INITIAL PERSONAL INTERVIEW. 

C. SURVEY QUESTIONNAIRE...ZZ!l 7 

1 . Target Audience.. 

2. Survey Sample Size.. 

3. Survey Design.20 

D. SECONDARY INTERVIEWS... 21 

E. SUMMARY..""’Zl 

TV. DATA PRESENTATION AND ANALYSIS.23 

A. INTRODUCTION.!!."" *!..23 

B. GROUP COMPARISON...*.^24 

C. RESPONSES TO DEVELOPMENTAL FUNDING AND SALES 

QUESTIONS___24 

1. What Products Are Going to Market?....................................................25 

2. How do SBIR Firms Approach Marketing?..._________26 

3. SBIR Product Consumption. 27 

4. Average Total Sales.28 

D. RESPONSES TO OTHER COMPANY AND PROJECT RELATED 

INFORMATION.29 

E. PROGRAM PERCEPTIONS FROM SBIR PHASE II FIRMS_30 

1 . Statement 25a Graphic Representation.31 

2 . Statement 25a Narrative Analysis.31 

3. Statement 25b Graphic Representation. 33 

4. Statement 25b Narrative Analysis... 33 

5. Statement 25c Graphic Representation. 35 

6 . Statement 25c Narrative Analysis.36 

F. SBIR PROGRAM MANAGER INTERVIEW.!!!."36 

G. SBIR FIRM RECOMMENDATIONS FOR IMPROVEMENT_ 37 

H. SURVEY DESIGN FEEDBACK. 38 

I . SUMMARY.... 

V. CONCLUSIONS AND RECOMMENDATIONS.41 

A. INTRODUCTION. 41 

B. CONCLUSIONS. ‘.”41 

C. RECOMMENDATIONS...!!!!!!.."!.41 

1 . Minimize the Emphasis on Commercialization.42 

2 . Conduct Regular Program Surveys.42 

3. Use a Small Survey Population to Prepare Program Results.42 

4. Modify the Survey Instrument to Enhance Responsiveness. 43 

D. ANSWERS TO RESEARCH QUESTIONS.. 43 

1. Primary Research Questions. 43 

2 . Subsidiary Research Questions.............. 43 

E. AREAS OF FURTHER RESEARCH_ .....45 

APPENDIX A - SBIR SURVEY INSTRUMENT_ 47 

viii 













































APPENDIX B - PROPOSED SBIR SURVEY INSTRUMENT.55 

APPENDIX C - RAW DATA TABULATIONS.62 

LIST OF REFERENCES.83 

INITIAL DISTRIBUTION LIST.85 


IX 







THIS PAGE INTENTIONALLY LEFT BLANK 



LIST OF FIGURES 


Figiire 4.1 Distribution of Intended Commercial Product.25 

Figure 4.2 Marketing Staff Hiring Strategy.26 

Figure 4.3 Customer Sales by Market.27 

Figure 4.4 Average Total Sales.28 

Figure 4.5 Probability of a Firm Independently Researching a SBIR Topic.30 

Figure 4.6 Frequency of Response to Statement 25a.32 

Figure 4.7 Frequency of Response to Statement 25b.34 

Figure 4.8 Frequency of Response to Statement 25c.35 


XI 











THIS PAGE INTENTIONALLY LEFT BLANK 


XU 



LIST OF TABLES 


Table 4.1. Distribution of Survey Respondents 


Xlll 




THIS PAGE INTENTIONALLY LEFT BLANK 


XIV 



LIST OF ACRONYMS 


AMC 

Army Material Command 

ARO-W 

Army Research Office - Washington 

DoD 

Department of Defense 

GAO 

General Accounting Office 

GPRA 

Government Performance and Results Act 

NSF 

National Science Foundation 

PEO 

Program Executive Office 

PM 

Program Manager 

RDEC 

Research Development and Engineering Center 

SBIR 

Small Business Innovation Research 


XV 


THIS PAGE INTENTIONALLY LEFT BLANK 


XVI 


ACKNOWLEDGMENTS 


I would first like to acknowledge the assistance of the Army Small Business 
Innovation Research Office. This work would not have been possible without their 
insight and support. 

I would like to thank: 

Professor Jeff Cuskey for his guidance and patience during the work in 
performing this investigation 

Doctor Haga for the very dry wit, humorous days in class, and superb guidance. 

My mother, grandmother, and wife for providing understanding, encouragement, 
and lifelong support. Their proding and loving support has been appreciated throughout 
the drafting of this Thesis. 

Ms. Sue Coates and the entire California Procurement Training & Assistance 
Center staff for their support and encouragement during this process. 

Finally, thanks to the business men and women who took the time to respond. 


xvii 



TfflS PAGE INTENTIONALLY LEFT BLANK 


xvm 



1. INTRODUCTION 


A. GENERAL 

“The relative weight that should be given to [agency] goals when 
evaluating proposals remains unclear.” (Ref. 1: p.7) 


The Small Business Innovation Research (SBIR) program provides significant 
opportunities for small businesses to participate in Federal Government research and 
development efforts every year. The program awards contracts to small companies that 
demonstrate in a competitive proposal process that they can perform innovative research 
and development that serves a Department of Defense (DoD) need. It also seeks to fund 
efforts that provide important commercial applications that add to national economic 
objectives. The method DoD uses to accomplish this goal is to provide funding up to 
$850,000 directly to each award wiimer for a specific research topic. 

Since its inception in 1982, the'program has evolved to reflect a compilation of 
agency goals indicative of a public program with multiple stakeholders. This project will 
elucidate how those goals impact program execution. 

1. The Problem 

All Federal agencies with extramural budgets for research and development are 

required to set aside 2 1/2 % of their total research/research and development funds to 

support the SBIR program. The presenting problem that this thesis will address is that 

the allocation of precious resource funds expended on this program are not adequately 

being accounted for in terms of results in accordance with the Government Performance 

Results Act of 1993. Without a full accovmting of benefits derived from the program, it is 
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impossible to quantitatively assess whether the program is successful in meeting the 
stated goals of the principle stakeholders. 

2. Potential Solutions 

The researcher proposes three key changes to the Army Research Office’s 
standard operating procedures; 1) Reduce its emphasis on outside investment as an 
evaluation criterion; 2) Conduct regular program surveys and ; 3) Use small survey 
populations to develop annual program results reports. 

3. What Happens ifthe Problem Remains Unanswered? 

If the program continues to exist in its current form without the controls or 
feedback loops needed to assess program success, the difficulty in determining a return 
on investment for the allocation of Federal research and development fimds will continue. 
This potential misallocation of resources will produce an opportunity cost to the taxpayer 
and reduce Government efficiency. 

B. OBJECTIVES 

The objectives of this project are to determine if a correlation between technology 
sales and commercialization potential exist to support the Army’s current evaluation 
criteria and to measure the prospects of commercial sales on the technical approach taken 
by prospective firms. It also seeks to validate whether current procedures are effective in 
satisfying agency goals and objectives and fulfilling Congressional intent. The 
completed thesis will build upon an earlier body of research conducted on projects fi'om 
1994-1996 and address public law, acquisition principles, and agency specific strategic 
plans. 

C. RESEARCH QUESTIONS 

The following research questions will be answered by the thesis. 
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1. Primary Research Question 

How does the Army define and measure Small Business Innovation Research 
(SBIR) Program success? 

2. Subsidiary Research Questions 

• What are the Small Business Innovation Research (SBIR) and Fast Track 
Programs? 

• What are the stated goals of the DoD SBIR Program? 

• What are the various laws, regulations, and policies that might affect the SBIR 
and Fast Track Programs? 

• What major products are created through the Army SBIR program and how 
does the Department of Defense integrate those products into its inventory? 

• How does the Department of the Army track the commercialization of SBIR 
products through the program’s life cycle? 

• Do current Department of the Army requirements generation, marketing, 
solicitation, proposal evaluation, source selection and surveillance procedures 
lead to enhanced program success and greater commercialization of products? 

• What changes are needed to the Department of the Army’s SBIR Program to 
enhance the program’s performance as it relates to stated goals? 

D. SCOPE, LIMITATIONS, AND ASSUMPTIONS 

1. Scope 

The scope of this thesis will include: (1) an in-depth review of the SBIR program 
and its stated goals, (2) an evaluation of the current proposal selection process, and (3) an 
objective analysis of whether the current system is optimizing program execution. The 
thesis will conclude with proposed reconunendations for optimizing program execution. 

2. Limitations 

The research will be limited by the fact that the Army SBIR Program has not 
undergone any exhaustive analysis of its effectiveness in producing outcomes. 
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Furthermore, due to the limited resources, feedback mechanisms for ascertaining 
conclusive commercialization results via product or process tracking for all Phase n 
participants are not available. 

3. Assumptions 

The following assumption will be used in preparing the thesis report. 

a. Any reader of the thesis is assumed to have a fundamental 
understanding of the Defense Department and Army in general, and at least a surface 
familiarity with the procedures and functions associated with the acquisition commimity. 

b. It is assumed that the provisions of the Small Business Innovation 
Research Program Reauthorization Act of 1999 will remain in force until 2008. 

c. It is assumed that the Army SBBR. Program’s standing operating 
procedures will remain fundamentally unchanged for the foreseeable future. 

E. DEFINITIONS AND ABBREVIATIONS 

A complete list of pertinent acronyms is provided in the Acronym listing in the 
front of this document. Several key terms, however, are shown below. 

1. Research and Development 

Any activity that is: (Ref 2) 

(a) A systematic, intensive study directed toward greater knowledge or 
understanding of the subject studied. 

(b) A systematic study directed specifically toward applying new knowledge to 
meet a recognized need. 
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(c) A systematic application of knowledge toward the production of useful 
materials, devices, and systems or methods, including design, development, and 
improvement of prototypes and new processes to meet specific requirements. 

2. Commercialization 

The process of developing markets and producing and delivering products for sale 
(whether by the originating party or by others), as used here, commercialization includes 
both Government and commercial markets. (Ref 2) 

3. Fast Track 

An expedited proposal evaluation process that requires third party investment for 
participation. This program affords research firms that arrange for third party investment 
to experience significantly higher chances of contract award by providing opportunities 
to leverage matching Federal funds of between $1 and $4 for every $1 the investor 
provides. (Ref 2) 

4. Extramural Budget 

The sum of the total obligations of research/research and development minus 
amounts obligated for Research/research and development activities by employees of the 
agency in or through Government-owned, Government-operated facilities. (Ref 2) 

5. Funding Agreement 

Any contract, grant, or cooperative agreement entered into between any Federal 
agency and any small business concern for the performance of experimental, 
developments, or research work funded in whole or in part by the Federal Government. 
(Ref2) 
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6. Subcontract 


Any agreement, other than one involving an employer-employee relationship, 
entered into by a Federal Government funding agreement awardees calling for supplies or 
services required solely for the performance of the original funding agreement. (Ref 2) 

7. Small Business Concern 

A small business concern is one that, at the time of award of Phase I and Phase n 
funding agreements, meets the following criteria: (Ref 2) 

(a) Is independently owned and operated; is not dominant in the 
field of operation in which it is proposing; has its principal place of business located in 
the United States and is organized for profit. 

(b) Is at least 50 percent owned, or in the case of a publicly owned 
business, at least 50 percent of its voting stock is owned by United States citizens or 
lawfully admitted permanent resident aliens. 

(c) Has, including its affiliates, no more that 500 employees. 
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11. BACKGROUND 


A. GENERAL 

In order to facilitate the discussion, we must first explore the events leading to the 
formulation of the SBIR Program. This chapter will specifically address the introduction 
of the Small Business Innovation Development Act of 1982; the key provisions imposed 
by the Act; significant developments fostered through subsequent Acts; program goals; 
the organization structure and mission of the Army Research Office; and a description of 
the Fast Track Program. 

B. SBIR PROGRAM INTRODUCTION 

In an effort to support the inclusion of small business initiatives in 
research/research and development, Congress established the Small Business Innovation 
Development Act in 1982 (Ref 3) to provide increased opportunities for small businesses 
to participate in research/research and development, to increase employment, and to 
improve U.S. competitiveness. 

The SBIR program was scheduled to expire on October 1, 1988. However, 
Congress initially extended the program to September 30, 1993. The reasons given for 
extending SBIR were that the program creates new jobs, increases productivity and 
economic growth, helps combat inflation, and stimulates exports. (Ref 4, p.3) In 
extending the program. Congress acknowledged that small business had not been 
receiving a fair share of Federal research/research and development dollars. In 1992, 
Congress enacted the Small Business Research and Development Enhancement Act 
which again extended SBIR, this time through October 1, 2000. President Bush signed 
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the bill into law on October 28, 1992. The current authorization (Ref 5) reauthorizes the 
program imtil September 30,2008. 

C. KEY PROVISIONS OF THE ACT 

Federal Agencies with research and development budgets exceeding $100 million 
are required to set aside a portion of those funds to support the program. The minimum 
funding level has steadily increased with each successive reauthorization. The original 
bill required 1% of an agency’s budget. Since then, the proportion has grown from 
1.25% during 1987-1992 to 1.5% in 1993 and 1994 to 2% in 1995 and 1996. Since 1997, 
at least 2.5 percent of all extramural research funds exceeding the $100 million threshold 
have been set-aside for SBIR. (Ref 6, p.46) 

D. SIGNIFICANT DEVELOPMENTS 

In 1992, Congress reauthorized the act with significant modifications. The Small 
Business Research and Development Enhancement Act of 1992 applied a 
commercialization” standard to the proposal selection process to encourage the 
promulgation of research with commercial market potential. The act defined 
commercialization as the process of developing markets and producing and delivering 
products for sale. (Ref 7) This major shift in focus altered how proposals would be 
evaluated in the future by placing a premium on a factor that, at times, influences the 
technical approach proposed by potential offerors. 

A second major change within the framework of the program took place with the 
1999 reauthoiization. The Small Business Innovation Research Program Reauthorization 
Act of 1999, added reporting requirements that supported the Government Performance 
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and Results Act of 1993. It specifically mandated that agencies include as part of their 
annual performance plan: (Ref 8) 

a) The establishment of performance goals to define the level of performance to 
be achieved. 

b) The expression of such goals in an objective, quantifiable, and measurable 

form. 

c) The establishment of performance indicators to be used in measuring outcomes 
from each program activity. 

d) A benchmarking system for comparing actual program results with established 
performance goals. 

e) A method to describe the means to be used to verify and validate measured 

values. 

These reports are forwarded to the Committee on Small Business of the Senate, and the 
Committee on Science and the Committee on Small Business of the House of 
Representatives. 

E. TOPIC GENERATION 

At least six months prior to the release of the SBIR solicitation in May of each 
year, Army Research Office-Washington (ARO-W) issues a call for topics to the Labs 
and Research Development and Engineering Centers (RDEC) to be considered for 
inclusion in the solicitation. Each LAB or RDEC is allocated a number of topics based 
on current Army guidance and expected future SBIR budgets. Topic authors must be 
Army civilian or Army military personnel with requisite technical expertise and be 
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assigned to the cognizant Lab or RDEC or an associated Progr am Manager (PM) or 
Program Executive Office (PEO). The topic authors in the Labs and RDECs develop 
topics that address their respective organizations’ objectives. Topic authors should also 
coordinate with their relevant battle Lab to get the “user” perspective. F inall y the topics 

submitted must be in accordance with the Lab and RDEC Technical Director’s priorities. 
(Ref 9, p.lO) 

F. SBIR PROGRAM GOALS 

The Small Business Innovation Research Program Reauthorization Act of 1992 
forms the basis of the current program’s execution. In the accompanying House Report, 
Congress noted that the program was highly successful in integrating small business in 
Federal research. (Ref 4, p.l) It further noted that small businesses were producing 
innovative goods and services, and that the program was the “catalyst in the promotion of 
research and development” for the nation’s high-technology industries. (Ref 4, p.l) The 
report stressed the ability of small firms to capitalize on scientific and technical 
innovations and quickly transform those findings into new products and services thereby 
enhancing national economic growth. (Ref 4, p.4) The legislation addressed the 
congressional concern that although small businesses were the most productive source of 
significant innovation in the nation, their share of Federally funded research and 
development efforts was understated in terms of contribution. (Ref 7) 

Congress designated four major goals (Ref 6, p.46) in authorizing the program to: 

1) stimulate technological innovation; 

2) use small business to meet Federal research and development 

needs; 
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3) foster and encourage participation by minority and disadvantaged 
persons in technological innovation; and 

4) increase private-sector commercialization of innovations derived 
from Federal research and development. 

G. SBIR PROGRAM PHASES 

The SBIR program is subdivided into three distinct phases. Although each 
agency participating in the program has the autonomy to administer the program 
differently, the basic parameters which define each phase are essentially the same. For 
illustration, the researcher will outline the phases as they relate to Army program 
execution. 

1. Phase I 

Phase I is a feasibility study of approximately six months in duration. Only 
proposals which respond to specific Army topics pubhshed in the second of two annual 
DoD SBIR Program Solicitations are eligible for Phase I consideration. Fxmding for 
Phase I is not to exceed $70,000, with an additional $50,000 available as a separately 
priced option if that provision it is included with the initial proposal. The option may be 
exercised, at the Army’s discretion, to fimd start-up Phase n activities for those Phase I 
projects which have been selected for Phase n negotiation and award. (Ref 9, p.4) 

2. Phase II 

There is no solicitation for Phase II proposals, but successful Phase I firms may 
be invited by the Army to participate in Phase n of the SBIR Program where the primary 
R/R&D work is conducted. Phase n awards are two year efforts that are funded 
incrementally up to a maximum of $750,000. (Ref 9, p.4) 
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3. Phase III 


Successful Phase m projects result in technologies, products, or services that can 
be marketed to Government or commercial customers outside the SBIR Program. This 
commercialization phase is the ultimate goal of each SBIR effort. By law. Phase m 
efforts can not incorporate SBIR funds. (Ref 9, p.5) 

H. ARMY RESEARCH OFFICE ORGANIZATIONAL STRUCTURE 

1. Organizational Structure 

The Army Research Office-Washington (ARO-W), a subordinate office of the 
Army Research Office, is the parent organization of the Army’s SBIR effort. It is 
structured with two distinct functions: ARO-W (A) manages the SBIR program and 
ARO-W (B) monitors and participates in DoD-wide science and technology activities 
that leverage the Army's investment in basic research. 

2. Mission 

The mission of ARO-W (A) is to serve as the liaison between ARO Headquarters 
and Army Materiel Command (AMC). The office’s primary function is to execute 
Army-wide technology development in partnership with industry and academia through 
three unique programs: The Army Small Business Innovation Research (SBIR) Program, 
the Army Small Business Technology Transfer (STTR) Program, and the Advanced 
Concepts and Technology Program (ACT II). Each program has rniique goals and 
operating parameters. The ARO-W acts as the executive agent for all Army SBIR 
activities and coordinates program actions within the Army's laboratories and RDECs. 
(Ref 10) 
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3. Proposal Evaluation 

Proposal submissions are evaluated on scientific, technical, and commercial merit. 
Because of the proprietary nature of the criteria and associated weighting, that 
information will not be disclosed as a component of this report. The researcher 
acknowledges that commercial potential may have a significant impact upon approval of 
Phase 11 awards. 

4. SBIR Marketing Assistance 

The Army SBIR office expends considerable effort to publicize program 
innovations. They assist firms in marketing their products through the annual SBIR 
Phase III success booklet publication. This booklet is distributed throughout the Army; at 
National, Regional, and State-level SBIR Conferences and workshops. In many 
instances, representatives of large businesses and venture capital firms attend such events 
to “network” with Army SBIR firms. SBIR Phase III success stories are posted on the 
ARO-W website and are routinely provided to SBA and in response to Congressional- 
related request. 

I. FAST TRACK PROGRAM 

The DoD initiated the Fast Track prograrii in an effort to improve the rate of SBIR 

commercialization. Starting with the 1996 solicitations, DoD initialed a two year pilot 

policy - the SBIR Fast Track - under which SBIR projects that attract matching funds 

fi'om third-party investors have a significantly higher probability of SBIR award. 

Additionally, the SBIR program office expedites proposal evaluation and processing to 

reduce the delay in reaching the market. The purpose of the Fast Track policy is to 

concentrate SBIR funds on those research and development projects most likely to result 

in viable new products that DoD and other will buy. (Ref 6, p.52) 
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At the conclusion of the initial pilot period, the Under Secretary of Defense for 
Acquisition and Technology extended the Fast Track pilot for two additional years, 
because of the promising early results. The Under Secretary also directed an independent 
analysis of Fast Track. The National Research Council (NRC) was asked to conduct that 
analysis. (Ref 6, p.51) The study’s focus covered a total of 379 DoD projects selected 
from among the 2,574 that received Phase II awards during 1992-1996. The sample 
contained award data from every agency operating under the DoD organizational 
structure. 

The NRC analysis of results concluded that the Fast Track program was clearly 
outperforming DoD Control Group awards. The areas of sales, additional developmental 
funding and expected sales were all sighted as having higher aggregate values. (Ref 6, 
p.44) 

J. PRIOR STUDIES 

In 1991, the General Accounting office (GAO) conducted a study across all 
Federal agencies participating in the program to evaluate the aggregate commercial 
trends of products that were studied and funded through Phase II. The 1991 survey 
questionnaire was sent to all Phase II awardees from the first four years of Phase II 
program awards: 1984-1987. The GAO selected the earliest recipients because studies by 
technology experts concluded that the incubation period for a concept to progress to a 
commercial product was five to nine years. (Ref 6, p.51) 

K. GOVERNMENT PERFORMANCE AND RESULTS ACT 

Congress passed the Government Performance and Results Act (GPRA) as part of 
a legislative framework to instill performance-based management in the Federal 
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Government. The Results Act established a management system to set agency goals for 
program performance and to measure results against those goals. In enacting the Act, 
Congress and the administration realized that the transition to result-oriented 
management would not be easy. For that reason, the Act provided for a phased approach 
to implementation. (Ref 11) 

Implementing the GPRA in a research environment is particularly challenging. In 
the past, research agencies have cited numerous barriers to their efforts to establish 
results-oriented goals and measures. (Ref 12) The barriers include problems in obtaining 
data to demonstrate results, accoimting for factors outside of the agency’s control that 
affect results, and dealing with the long time periods often needed to see results. 

Over the past several years, the Government Accounting Office has issued reports 
that identified practices for improving GPRA implementation in Federal agencies. These 
reports have focused on, among other things, overcoming agency specific barriers, 
improving the usefulness of annual performance plans, and measuring program results 
that are under limited Federal control. These reports point out the depth and scope of 
management practices needed to successfully implement performance-based management 
as envisioned under the GPRA. (Ref 11) 

L. SUMMARY 

This chapter provided essential backgroimd information on the SBIR Program 
designed to enhance the readers’ understanding. It focused primarily on the origins of the 
program, significant developments, and basic structural fi-amework. The chapter 
concluded with a short synopsis of the Government Performance and Results Act and its 
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influence on reporting results. The following chapter will discuss the research 
methodology used to gather data for analysis. 
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III. RESEARCH METHODOLOGY 


A. GENERAL 

The research methodology used to gather data for the thesis centered on two 
primary means: personal interviews with the program manager and program executors 
and a survey questionnaire focused on previous award recipients. Chronologically, there 
was an initial interview that launched the research process, followed by the transmission 
of the survey questionnaire, and then the conduct of an additional personal interview with 
the Army Research Office SBIR Program Manager. Each of these steps is discussed 
below. 

B. INITIAL PERSONAL INTERVIEW 

The purpose of the initial interview was to ascertain whether an objective analysis 
of the SBIR Program was feasible and to determine how the Army Research Office was 
obtaining, assessing, and reporting the performance of contractors that had participated in 
the program. The goal of this phase was to set the direction for further research 
exploration. 

During the initial interview, (Ref 13) a program support contractor supporting the 
Army Research Office SBIR effort, provided his insight. He was instrumental in 
assessing the state of the program and offering a unique perspective into the program that 
shaped the direction of the project. 

C. SURVEY QUESTIONNAIRE 

The survey was conducted draw a parallel between prior studies and current 
practice. The goal was to produce a baseline for comparison that could be used to 


measure program results attributable to previous changes in organizational practices. The 
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method for selecting the survey recipients (target audience), and constructing the actual 
questionnaire (survey design) are presented in the following paragraphs. 

1. Target Audience 

Given the popularity of the SBIR Program, the researcher decided that the best 
strategy for conducting the survey would be to analyze Phase II awards for both Fast 
Track and non-Fast Track participants in a given period. A target audience was selected 
using only those firms that received a Phase II award from 1996 through 1998. The 
researcher determined that only those firms that had matriculated through Phase II would 
be able to provide any return on investment for the taxpayer via commercialization of 
technology derived from the program. The critical factor in pursuing such a focus was to 
preempt any claim of sampling error or survey variance by surveying a statistically 
significant number of firms. 

The target audience analysis rationale was directly attributable to program 
guidelines that allow the ARO to invite only selected firms to negotiate Phase I interim 
and Phase II awards. (Ref 14) Phase I interim awards were excluded because of 
database constraints that limited their validity. The 1996 date reflected the last fiscal year 
in which SBIR program research proposals were evaluated in the National Science 
Foundation study. (Ref 6) Finally, the latter date represented the last year a firm would 
have had to begin research in order to complete the Phase II effort before moving into 
Phase III. This cutoff date was further supported by analysis of the so-called two-year 
“incubation” period after Phase II completion that firms experience before any 
innovations can be brought to the marketplace. (Ref 15) 
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A search of the SBIR Award Database at the Office of Small and Disadvantaged 
Business Utilization for the Office of the Secretary of Defense served as the starting point 
for identifying specific survey participants. (Ref 6) The researcher identified 324 Phase 
II awardees that fit the target audience profile. A sample population of 65 firms was 
randomly selected fi-om the SBA Technet database for the study. An attempt was then 
made to telephonically contact each firm identified on the list in order to determine two 
points: 1) if the firm was still in existence and 2) to ascertain who in the firm would serve 
as the point of contact for the questionnaire. The researcher hoped that by phoning each 
firm, locating the original researcher listed in the database, and explaining the concept of 
the project before sending the questionnaire would serve to raise the response rate. Of 
the 65 firms on the initial target audience list, 49 firms (75%) were contacted and 
identified as survey recipients. Sixteen firms (25%) identified on the target audience lists 
were deleted due to bankruptcy, restructuring, loss of internal documentation, merger, or 
an unwillingness to participate. Of the 49 surveys transmitted, 37 surveys were returned 
yielding a response rate of 76%. 

2. Survey Sample Size 

Given the responsiveness noted above, it is appropriate to ask whether the sample 

was large enough so as to mitigate or eliminate any potential clams of sampling error. In 

other words, was the survey sample of sufficient size to provide a clear and reliable 

indication of the responses that would have been given if the entire Phase II award 

population had been surveyed? Johnson (Ref 17) states that “when [a] sampled 

distribution lacks symmetry, n (the number of observations) may have to be quite large 

(maybe 50 or more) before the normal distribution provides a satisfactory approximation” 

for the probability distribution of the mean. For the analysis contained in the next 
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chapter, the author assumes that the responses presented for analysis in Chapter IV are 
normally distributed. 

The researcher used a combination of factors to determine the minimum required 
sample size using the largest standard deviation for the three scaled sub-components 
contained in the survey. When combined with a maximum desired error factor of 
+/-.30 points from the sample mean, a confidence coefficient of 80% (1.65), and standard 
deviation for the recorded responses of 1.0836, the computation yielded a sample size of 
36 completed surveys. A total of 37 surveys were collected and analyzed. 

3. Survey Design 

The researcher had two concerns when constructing the smvey questionnaire. 
The first was to design a survey that produced objective results that would allow for 
statistical analysis of the SBIR Program and commercialization rates. The second was to 
pose the questions/statements in such a way as to eliminate potential bias in the results. 
A survey questionnaire, as distributed to the target audience, is contained in Appendix A. 

Given the concerns of the researcher, the questionnaire was designed with three 
sections. The first section addressed specific historical data for each contract the firm 
imdertook. The purpose of this was to confirm that the information listed on the DoD 
Under Secretary of Defense Website database matched the SBA Technet database. The 
concept was to provide preliminary check of the validity of the information contained in 
the database to ensure that the comments and responses provided by the firm were 
credible. 

The second section aimed to gather additional information that would facilitate 
further statistical analysis through the correlation of data derived from DoD SBIR 
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records. The final section’s focus was to gain an understanding of each firm’s 
perspective. It specifically sought to gain insight into how each firm was approaching the 
program and how various approaches were influenced by the firm’s beliefs. 

D. SECONDARY INTERVIEWS 

Secondary Interviews were conducted two months after survey initiation. The 
purpose of the secondary interviews was to assess what policies key decision makers 
deemed essential to qualifying the results of the program. The underlying intent was to 
ascertain whether they viewed a reporting mechanism as necessary given the 
overwhelming Congressional support for the program. 

E. SUMMARY 

This chapter provided an in-depth discussion of the research methodology used to 
develop the survey instrument used to gather information for the thesis. It identified the 
purpose of the initial interview and the rationale supporting survey instrument 
development. Additionally, it identified the need for an assessment of the program from 
a policy-making perspective to ascertain how program results were viewed from a top- 
down perspective. The next chapter will present a statistical examination of the collected 
data and the siunmarized results of both sets of interviews. 
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IV. DATA PRESENTATION AND ANALYSIS 


A. INTRODUCTION 

T his chapter presents the results of a survey administered to SBIR Phase II 
participants. The target audience’s responses are displayed in graphical form for ease of 
interpretation. The data and subsequent analysis presented in this section will illuminate 
how funding and program solicitation dynamics impact program results. This chapter 
will also discuss the economic impact of program outcomes for technologies and 
products developed as a direct result of the SBIR program. 

The SBIR research survey instrument was based on a previous survey that the 
GAO conducted in 1992 and a National Science Foundation Paper. (Ref 6) New 
questions for the survey instrument were developed as a result of dialog between the 
researcher and the Army Research Office as well as from conversations with current 
SBIR Phase II participants. The final survey was reviewed and subsequent changes made 
with the assistance of a survey instrument expert at the Naval Postgraduate School. (Ref 
18) The final survey reflected high face validity between intended and perceived 
meaning of the questions. 

The SBIR survey was sent to 49 Phase II program participants during a three- 
month period starting February 2001. The survey yielded a 76% percent response rate 
with 37 firms meeting the feedback closing date. Participation was voluntary with the 
condition that individual survey responses would be strictly confidential. Table 4.1 
represents the distribution of survey respondents. As far as can be determined, the survey 
sample appears to be representative of the entire award population. 
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Fiscal Year of 
Award 

Target Audience 

Survey Respondents 

Percentage of 
Respondents 

FY96 

16 

12 

75% 

FY97 

19 

16 

84% 

FY98 

14 

r 9 

r 64% 

Total 

49 

37 

76% 


Table 4.1. Distribution of Survey Respondents. 
(Source: Designed by researcher) 


B. GROUP COMPARISON 

The Army SBIR contract awards represent a cross section of awards given over a 
three year period from 1996 to 1998. The firms selected for participation were rando ml y 
chosen from the 324 contract awards negotiated during that period. 

The survey data collected was analyzed against two groups for a basis of 
comparison. The first group of awards, 1996 DoD Fast Track award winners, consisted 
of firms that qualified under the Fast Track program for enhanced commercial potential. 
This group received special evaluation consideration and expedited proposal processing. 
These awards represented the Tjest in class because of an optimal combination of sound 
technological approach, researcher credentials, and commercial potential as validated by 
third party investment funding. 

The DoD control group represented a sample of all SBIR contracts across the 
DoD spectrum for fiscal 1996. They embody all of the salient characteristics of the 
research projects the program seeks to fund and constitute a baseline of what the average 
award should be. 

C. RESPONSES TO DEVELOPMENTAL FUNDING * ND SALES QUESTIONS 

In this survey, firms were asked to provide quantifiable business outcomes as a 
direct result of their specific project. Developmental funding and sales questions asked 
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firms to describe the extent to which they were offering products developed under the 
program to the commercial market. 

1. What Products Are Going to Market? 

The SBIR program affords firms the opportunity to produce a myriad of products 
and services through innovative research. Figure 4.1 displays the types of products each 
surveyed group produces. 




DoD Fast Army I>oD 

Track SBIR Control 

Awards Group 


□New or Improved 
Service Capability 

□Process Technology 


!0 Final Hardware 
Product 

IH Intermediate Hardware 
j Product or Component 

iBSoftware 


; BNo Commercial 
Applications Planned ■ 


Figure 4.1 Distribution of Intended Commercial Product. 
(Source: Designed by the researcher) 


When compared against Fast Track awardees and the DoD Control Group, the 
Army SBIR group is heavy weighted towards software applications and final hardw^are 
products. This phenomenon is indicative of two key themes: a solicitation process that 
promotes material solutions to army requirements and a continuous push to digitize the 
force. 
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2. How do SBIR Firms Approach Marketing? 

A marketing strategy can positively or negatively affect a firm’s ability to move a 
product to market. Some of the best innovations “die on the vine” because they have sat 
idle on the shelf without a mechanism to move from the developer to the customer. 
Completing research and marketing a product can be a challenge for some firms because 
of the disparate skill sets needed to complete marketing tasks. Figure 4.2 illustrates how' 
firms approached marketing from a company perspective with regard to staff hiring 
practices. 


JENot Needed 
^Completed 
^ Undeway 
■Planned 


DoDFast Anny SBIR DoD Control 
Track Awards Group 

Figure 4.2 Marketing Staff Hiring Strategy 
(Source: Designed by the researcher) 

Of the 37 Army SBIR firms surveyed, 17 decided that there was a need for a 
marketing staff to facilitate interest in their product. Although this closely mirrors 78% 
of Fast Track firms, the difference between those two groups can be found in the 
percent^e of firms having actually started the hiring process. Fast Track firms are much 
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more deliberate in their efforts to bring in expertise when die need is identified. The 
percentage of finns that have started or completed the marketing staff hiring process are 
57% and 24% for Fast Track awardees and Army SBIR firms respectively. This is most 
likely attributed to the influence of the oulside investment needed to qualify for Fast 
Track consideration. Convereeiy, Army SBIR firms move at a much slower pace. This 
may serve to hinder a firm’s ability to realize commercial market sales. 

3. SBIR Product Consumption 

An analysis of Army SBIR sales data conclusively demonstrates that Army- 
proposals are developed with a narrow focus on military applications. As a group, DoD 
purchased 85% of the total sales generated from Army SBIR funded technology. Figure 
4.3 depicts sales by sector of the three groups and clearly reveals that DoD is the prime 
beneficiary of Army SBIR program objectives. 

100 % 

80 % 

60 % 

40 % 

20 % 

0% 

DoD Fast Army DoD 
Track SBIR Control 

Awards Group 

Figtire 4.3 Customer Sales by Market 

(Source: Designed by the researcher) 
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4. Average Total Sales 

Average total sales data per award is a key performance metric. This figure 
represents an empirical measure that can assist program managers in objectively 
assessing program success via a return on investment analysis. Figure 4.4 graphically 
depicts how Army SBIR awards have performed against Fast Track awardees and the 
DoD Control Group. 


I SI40.000 
: $ 120,000 
$100,000 
$80,000 
$60,000 
$40,000 
$20,000 
$0 . 


Figure 4.4 Averse'Total Sales 
(Source: Designed by the researcher) 

The exceptional performance of Army SBIR firms is an interesting outcome of 
the research. As a group. Army SBIR fimis outperformed DoD Fast Track firms by 21% 
and the DoD Control Group by 164% per average total sales per award. These findings 
are particularly impressive given that the DoD consumes 85% of all Army SBIR 
products. 



DoD Fast Track Army SBIR Awards DoD Control Group 
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Ten firms surveyed commented tiiat Army projects had clear linkages to Army 
and DoD requirements that created a ready market for their products. The availability of 
this market influenced nine of those ten firms to principally focus on rapidly developing 
their technologies to conform to DoD requirements first and then seek to exploit 
commercial opportunities. When this commitmait is coupled with the aforementioned 
DoD market dominance, they create a synergy that magnifies program return on 
investment. 

D. RESPONSES TO OTHER COMPANY AND PROJECT RELATED 

INTORMATION 

Fostering ground-breaking research is a key tenet of the SBIR program. Given 
the responses. Army SBIR firms overwhelmingly conveyed that they would have not 
dedicated resources and effort to study Army topics. As a group, 16 Army SBIR finns 
responded that they would “Definitely Not” have explored the topic they received 
funding to study. Another 13 stated that they would “Probably Not” have opted to 
explore their specific topic. Those two subgroups combined represented 80% of the 
responses. 

When compared against Fast Track awardees, the dijSerences between the groiq)s 
are significant. Only 13% of all Fast Track aw'ardees stated that they would “Definitely 
Not” or “Probably Not” have studied the topic for which they received funding. This fact 
is a clear indication that research firms see little commercial value in the technologies the 
Army wants researched. When asked why they would not have researched their specific 
topic, 11 firms commented that they believed the purpose of the research was to meet a 
specific Army research requirement that did not have any “true” civilian applications. 
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Figure 4.5 Probability of a Firm Independently Researching a SBIR Tc^ic 

(Source: Designed by researcher) 

E. PROGRAM PERCEPTIONS FROM SBIR PHASE H FIRMS 

In the final portion of the survey, firms were asked to provide their opinions of 
■various aspects affecting their overall perception of the program. A five-point Likert 
scale was used to formulate an ass^sment of how each firm felt about third party 
investment, technical approach, and marketing.- Under this approach, one was selected to 
represent the hipest score and five the lowest The questions asked each firm to 
determine the extent to which they agreed or disagreed with specific principles affecting 
their proposal’s ability to satisfy Army SBIR solicitation requirements. The scale used 
was as follows: 

1 - Strongly Agree 

2 - Agree 

3 - Neither Agree nor Disagree 

4 - Disagree 

5 - Strongly Disagree 




! 



1 


1 



^Definitely Not 
^Probably Not 
® Uncertain 
^Probably Yes 
* Definitely Yes 
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Scores closer to one (1) mean that responses displayed a high degree of 
association to the proposed statement Scores closer to five (5) mean that responses 
displayed a high degree of association to the alternative hypothesis. 

The summarized raw responses to each of the three statements in the Other 
Company and Project Related section questionnaire are provided on the following 
subsections. Each para^ph lists the statement posed to Phase n firms and presents a 
graphical depiction of the summarized refuses. The mean of the responses, the 
standard deviation, and the Confidence Interval (Cl) range are all described in detail 
below each graph. 

The scaled responses are interpreted for favoring or disfavoring the impact of 
investment funding on program participation and proposal submission. A narrative 
analj^is describes how the researcher interpreted the scaled responses. 

Statement 25a - “ I believe outside investment by a third party is proof of the 
commercial potential of a research project”. 

1. Statement 25a Graphic Representation. 

See Figure 4.6 on following page. 

2. Statement 25a Narrative Analysis. 

The rationale for developing this statement was predicated on whether or not 
outside investment truly provides insight into a firm’s ability to maiicet its potential 
products thereby enhancing prospective sales. If a firm agreed with the statement, then 
we could infer that the firm would place a significant value in marketing their products 
throughout the research process to realize a faster return on investment for any 
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forthcoming developments. If a firm disagreed, we could infer that they did not place 
value in obtaining outside investment fimding. 



Figure 4.6 Frequency of Response to Statement 25a. 
(Source: Designed by the researcher) 


Mean Score: 3.1351 

Standard Deviation: 0.822 

Cl Range: 2.9587-3.3116 

The pattern of the oral and written comments tended to support both sides of the 
argument. 15 of the 37 respondents that disagreed with the statement cited a belief that 
outside investment in a SBIR firm during Phase I and II constituted a “proposal strategy” 
that served to gain preferential treatment via inclusion in the Fast Track program during 
the contract evaluation phase. Key words such as “manipulation” and “temporary loans” 
were used when describing how competing firms could make proposals look more 
promising on paper. One firm provided a detailed example where people could make 
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clandestine agreements with the so-called “investors” on a business loan and later pay the 
loan back to the “investof ’ once the Phase II funding is approved. 

Four firms noted that audit procedures do not seem foolproof. In this situation, 
die “outside investment” is not truly an investment, it is simply a “temporary loan” to 
make it comply with Fast Track requirement, but in reality no “real” money is spent on 
the project. 

The opposing view supported by 10 of die 37 firms cited the commitment of 
currency as the only method to effectively determine if a concept was worth exploring. 
Although none of the firms would commit to an extreme view of “Strongly Agree” or 
Strongly Disagree”, their approach to this question dictated that deterroining a consensus 
on this issue w'^ould be difiScult 

Statement 25b - ‘T would change a superior technical research ^proach to a 
slightly lesser technical research approach if I could gain access to outside investment” 

3. Statement 25b Graphic Representation 

See Figure 4.7 on the following page. 

4.. Statement 25b Narrative Analysis 

The rationale for developing this statement was to'determine if outside investment 
shaped the proposed research approach. If a firm agreed with die statement, then we 
could infer that diey placed a greater value on potential commercial sales opportunities to 
non-DoD customers throu^ the development of dual use technologies. Disagreement 
woiild suggest that firms thou^t their optimal proposed solution could be compromised 
by outside investor influence. 
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Figure 4.7 Frequency of Response to Statement 25b. 

(Source: Designed by' the researcher) 

Mean Score: 2.7838 

Standard Deviation: 1.0836 

Cl Range: 2.5512- 3.0164 

The variance in responses for this technical approach question suggests that each 
firm decides a proposed course of action after careful examination of its oum business 
considerations. Once internal and external factors are evaluated, companies will have to 
pro\ide a balanced proposal that addresses both technical merit and the potential for 
commercialization. 

The key point for program executors is to determine how best to structure 
solicitations to mitigate the influences of this decision making process while completely 
fulfilling agency specific research requirements. This dilemma represents an inherent 
conflict within the program because awards are required to support research that meets 
the needs of the awarding agencies, yet products developed are also expected to be 
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successful in the commercial marketplace. Eight of the 14 firms fiiat reloaded with 
“Strongly Agree” or “Agree” cited this dilemma as a central tenant in their response. 
They indicated that business objectives forced them to utilize this ^proach as a means of 
maintaining competitiveness within their respective areas of expertise. 

Seven of the 12 firms surveyed responded that they disagreed with the statement 
because of their belief that outside investors would influence the process to such a high 
degree that the final proposal might be evaluated as more than “sli^tly less” than 
optimal. 

Statement 25c - “Raising the interest of outside investors for SBIR r^earch 
topics is challenging.” 

5. Statement 25c Graphic Representation. 



Strongly Agree Neither Disagree Strongly 
Agree Agree near Disagree 

Disagree 


Figure 4.8 Frequency of Response to Statement 25c. 
(Source: Designed by the researcher) 

Mean Score: 1.7297 

Standard Deviation: .9324 
Cl Range: 1.5296-1.9298 
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6. Statement 25c Narrative Analysis. 

The rational for developing this statement was that if respondents agreed with this 
assertion, then an argument could be made that Army SBIR topics were considered 
abstract and provided little commercial potential. Low levels of additional 
developmental funding and sales could potentially be subsequent outcomes of this 
reduced commercial potential. 

Based on the low mean score of 1.7297 and the written and oral comments, the 
researcher surmises that there is a strong belief within the SBIR firms that SBIR projects 
are too narrowly constructed to appeal to outside investors seeking to exploit commercial 
applications. Although the program is satisfying Army research objectives, the Hgtf i 
indicate that the SBIR program is continuing to fall short of commercial market 
expectations. 

Eight of the 32 firms that responded with “Strongly Agree” or “Agree” noted that 
only a very small commercial market existed for the technologies they received an award 
to study. Four firms noted persistent problems with attracting capital. This was 
attributed to low commercialization prospects and the perception of low profitability with 
DoD contracts. Finally, One firm noted that they had tried to meet with a large firm in 
the same industry and could not because of fears of patent issues. 

F. SBIR PROGRAM MANAGER INTERVIEW 

The researcher conducted a personal interview with Dr. Kenneth Bannister, Army 
SBIR program manager, to ascertain how key policy makers viewed program execution. 
The interviews were conducted during the April 2001 Spring SBIR Conference in Crystal 
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City, Virginia. The purpose was to determine what, if any, steps were necessary to 
optimize the SBIR process. 

When asked about sharing project results across the DoD/Govenunent spectrum, 
Banister commented that SBIR results were published in booklets and brochures that are 
subsequently distributed throughout the business cormnimity via conferences, 
organizational websites, and the Defense Technical Information Center (DTIC). (Ref 20) 
He believes the effectiveness of these efforts to publicize the benefits of the program are 
commensurate to those used by other industries to promote their innovative products. 

Bannister (Ref 20) acknowledged that reporting quantifiable successes remains a 
dilemma for there is no formal process to gain empirical evidence to support the degree 
of successful commercialization within the program. More importantly, he noted that 
such a requirement does not exist, either in statute or policy, for companies to provide 
such data. This failure to gather empirical evidence and a reliance upon anecdotal 
methods such as media stories and “word of mouth” may serve to hinder the 
organization’s ability to comply with the annual performance reporting requirements 
required by GPRA provisions and the 2000 SBIR program reauthorization. 

G. SBIR FIRM RECOMMENDATIONS FOR IMPROVEMENT 

The researcher foimd that this study provided valuable insight into the factors that 
affect proposal construction. One of the most illmninating sections of the survey was the 
follow-up dialog between the researcher and the surveyed firms. The Army is forUmate 
to have such a motivated base of research firms competing to provide ground-breaking 
technologies to solve emerging requirements. Some specific comments from firms 
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indicate opportunities for improvement within the program. Firms provided the 
following responses when asked how they would improve the program: 

“Reduce the focus of outside investment.” 

“Focus research on areas that lend themselves to commercial products,” 

“Increase the contract award amoimt to keep pace with business cost.” 

H. SURVEY DESIGN FEEDBACK 

Subsequent review of the data provided evidence that ten questions on the original 
survey instrument used in this study to be of little use. Overall, the firms surveyed were 
reluctant to reveal personal information about matters such as the number of founders, 
their backgroimds, and when the firm was started. Additionally, questions that involved 
estunates from technical personnel, such as expected sales, returned a series of vague 
responses that did not add value to the project nor would have any tangible impact on 
future program assessments 

I. SUMMARY 

This chapter presented the survey data and provided an analysis of their respective 
impacts on the program. Summarized results of key questions were provided along with 
a narrative interpretation of what the results mean in terms of sales, marketing strategy, 
customer type, and types of products produced. 

Based on the survey results it is clear that the program is successful in producing 
a return on investment that is substantially higher than the DoD average; that the DoD is 
the primary beneficiary of the Army SBIR research efforts; that research firms would not 
be involved in Army SBIR efforts if the program were to cease it existence; and that 
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solicitation evaluation procedures do impact each research firm’s approach to proposal 
development. 
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V. CONCLUSIONS AND RECOMMENDATIONS 


A. INTRODUCTION. 

“There is great room for upward change.” (Ref 26) 


This chapter provides the conclusions and recommendations for the data collected 
from SBIR participants. 37 research firms responded to 25 closed end questions. The 
results of the survey are used to answer the primary and secondary research questions 
outlined in Chapter I. 

B. CONCLUSIONS 

The Army is successfully achieving its SBIR program commercialization goals. 
The average sales per award clearly indicate that the Army projects surveyed are 
outperforming both DoD and Fast Track award averages. 

Firms receiving SBIR funding are producing useful products that fulfill DoD 
requirements. The DoD consumption of 85% of Army SBIR results indicate that Army 
SBIR solicitations are filling vital DoD research requirements that may not otherwise be 
satisfied 

C. RECOMMENDATIONS 

This survey provides several lessons about how firms view the SBIR program and 
how those views impact their approach to each solicitation. The following are 
recommendations to assist ARO-W (A) in capitalizing on each firms’ already strong 
desire to participate in the program. They also provide a mechanism for ARO-W (A) to 
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collect the requisite data to fulfill reporting requirements mandated by the Government 
Performance and Results Act. 

1. Minimize the Emphasis on Commercialization 

The empirical data clearly proves that DoD is the main consumer of Army SBIR 
technological developments. Reducing the commercialization eveiluation factor weight 
will encourage firms to provide the best approaches to Army research requirements. 

2. Conduct Regular Program Surveys 

The data to gauge program effectiveness is available, but a regular mechanism is 
needed to capture it. The collection of this data will be helpful in preparing anmig] 
performance reports for GPRA reporting requirements. An electronic version of the 
proposed survey instrument in Appendix B could be used to perform this task. 

Additionally, it is recommended that contracting officer technical representatives 
at Army Labs and RDECs conduct the survey. They will have a working relationship 
with the performing research firms that can enhance the quality of the collected data. 

3. Use a Small Survey Population to Prepare Program Results 

Because the program is relatively stable, a small sample size will be sufficient to 
achieve a level of certainty required to present a suitable projection of program benefits. 
The key point to remember is that this program accounts for only 2.5% of an agency’s 
extramural budget in excess of $100 million. Since we do not provide as much scrutiny 
to large businesses with respect to how well they have commercialized their Federally 
sponsored research funding, the level of confidence needed to fulfill the GPRA report 
requirement should weigh the estimated costs of data collection against the projected 
benefits derived fi-om such a report. 
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4. Modify the Survey Instrument to Enhance Responsiveness 

The researcher recommends modifying the survey instrument to reflect questions 
of greatest impact for assessing program success. Reducing the number of questions and 
simplifying the survey could enhance future response rates. A sample survey reflecting 
the questions of greatest impact is enclosed in Appendix B. 

D. ANSWERS TO RESEARCH QUESTIONS 

1. Primary Research Questions 

Hov^^ does the Army define and measure Small Business Innovation Research 
(SBIR) Program success? 

The Army defines program success as complying with DoD’s four stated goals; 
1) stimulating technological innovation, 2) using small businesses to meet Federal 
research and development needs, 3) fostering participation of minority and disadvantaged 
firms in technological innovation, and 4) increasing private-sector commercialization of 
innovations derived from Federal research. It further defines program success as funding 
research proposals that provide the best technological solutions to Army research needs. 

The relative measure for success is a function of funding all high quality 
proposals until program funds are fully expended. 

2. Subsidiary Research Questions 

• What are the Small Business Innovation Research (SBIR) and Fast Track 
Programs? 

The SBIR program is a set-aside effort that allows small businesses to participate 
in Federal research initiatives. The Fast Track program is an expedited proposal review 
process that provides preferential treatment to firms that secure outside investment to 
augment Government provided research funding. 
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• What are the stated goals of the DoD SBIR Program? 

The stated goals of the DoD SBIR program are to: 

a) stimulate technological innovation; 

b) use small business to meet Federal research and development 

needs; 

c) foster and encourage participation by minority and disadvantaged 
persons in technological innovation; and 

d) increase private-sector commercialization of innovations derived 
from Federal research and development. 

• What are the various laws, regulations, and policies that might affect the SBIR 
and Fast Track Programs? 

The Small Business Innovation Development Act in 1982 provided the foundation 
for the SBIR program. Successive reauthorizations via the Small Business Research and 
Development Enhancement Act of 1992 and 2000 have added the commercialization 
tenet that currently dominates the focus of the program. 

• What m^or products are created through the Army SBIR program and how 
does the Department of Defense integrate those products into its inventory? 

The Army SBIR program requirements cover the full spectrum of research 
solutions. Software applications, intermediate and final hardware components, process 
technologies, and new or improved service capabilities are the major products of the 
SBIR program. 

Products are integrated into the inventoiy through acquisition of the technology or 
final product. The Army seeks to accomplish this task by linking the contractor with the 
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Army Lab or RDEC responsible for developing the requirement. This linkage ensures 
that future related research efforts will have some intrinsic knowledge of past SBIR 
projects. 

• How does the Department of the Army track the commercialization of SBIR 
products through &e program’s life cycle? 

It does not have a formal mechanism to accomplish this task. The Army SBIR 
effort rehes on “word of mouth” success stories or media exposure to gauge Phase III 
success. 

• Do current Department of the Army requirements generation, marketing, 
solicitation, proposal evaluation, source selection and surveillance procedures 
lead to enhanced program success and greater commercialization of products? 

The answer is yes and no. Congress has determined that the program is 
successful in accomplishing its program goals, but there are no recurrent empirical 
measures through which the Army, or any other agency, can readily use to substantiate 
that statement. 

• What changes are needed to the Department of the Army’s SBIR Program to 
enhance the program’s performance as it relates to stated goals? 

A review of the evaluation factors to support superior technical approaches and to 
minimize the impact of outside investment is recoihmended to enhance program 
execution. 

E. AREAS OF FURTHER RESEARCH 

There are still several issues that can be addressed in further detail. While the 
data show that the program is clearly successful from a sales perspective, the ability to 
qiiantitatively measure how well other program goals are fulfilled is a problem. The 
researcher recommends further study to determine how program participation by 
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disadvantaged and minority firms could be increased. Additionally, further study is 
needed to determine ho^v Army solicitation requirements can be drafted to capitalize on 
dual use applications that could enhance the commercialization potential of future SBIR 
awards and products. 
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Army SBIR Award Survey Questions 

Proposal Title: 

Principle Investigator: 

Phase n Contract #: 

1. Is the information listed above correct? 

Yes. Go to Question 2 

No. Provide correct information below then proceed to Question 2. 

Title: 

Principle Investigator: 

Phase II Contract Number: 

2. What is the current status of this SBIR Project? (Check one) 

Project has not yet completed Phase II. 

Project completed Phase II. 

Project was not awarded a Phase II. 

_Don’t Know 

3. How do you expect to commercialize your SBIR award? (Check one) 

_No Commercial Product, Process, or Service is Planned 

_Software 

_Intermediate Hardware Product or Component 

_Final Hardware Product 

_Process Technology 

_New or Improved Service Capability 

_Don’t Know 


Question 4 concerns whether the project is still active or if and when it was dropped. 


4. When did your company drop the project? (Check one) 

Still Active, not dropped. 
Don’t Know 

More than one year after completing Phase II. 


During or at the end of Phase II 
Within one year after completing Phase II. 
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If the technology developed during this project has led to no additional developmental 
funding or sales (and neither of these is expected to occur), skip to Question 16. 


Additional Developmental Funding and Sales 

PLEASE READ THE FOLLOWING DEFINITIONS 

Additional Developmental Funds: Include funds from federal or private sector sources, or 
from your own company, used for further development of the technology develop during the 
Phase II project. 

Sales: All sales of a product, process, or service, to Federal nr Private Sector Customers 
resulting from the technology developed during this Phase II project. 

(Includes the sale of technology or rights, etc.) 


5. To date, what has been the total additional developmental funding for the technology 
developed during this project? 

(Do not include related SBIR funds receivedfrom DoD or other federal agencies. Enter dollars 
provided by each of the following sources. If none, enter zero.) 


Sources 

a. Non-SBIR Federal Fimds 

b. Your Own Company 

c. Other Private Company 

d. U.S. Venture Capital Institution 

e. Foreign Venture Capital Institution 

f. Private Investor 

g. Personal Funds 

h. State or Local Governments 

i. College or Universities 

j. Other Sources (Specify) 


Dollars 



Dt fi’tKnow 
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your company’s activities with other companies and investors? 

(Check all that apply) 


Ongoing Finalized 

Negotiations Agreements 

Activities I-■ ■-■ 

a. Licensing Agreement 

b. Sale of Complete Ownership _ _ 

c. Sale of Partial Ownership _ _ 

d. Sale of Technology Rights _ _ 

e. Joint Venture Agreement _ _ 

f Marketing/Distribution Agreement _ _ 

g. Manufacturing Agreement _ _ 

h. Other _ _ 

i. Don’t know I_I I_ 


7. Which of the following, if any, describes the type and status of marketing activities by your 
company and/ or your licensee for this project? 

(Check only one block for each activity) 


Marketing Activity Plaimed 

a. Preparation of Marketing 

b. Hiring of Marketing Staff 

c. Publicity/Advertising 

d. Test Marketing 

e. Other (Specify): 


Not Don’t 

Underway Completed Needed Know 



8. Has your company and /or licensee had any actual sales of products, processes, services or 
other sales from the technology developed during this project? (Check all that apply.) 


a. Sales of Product(s) 

b. Sales of Process(es) 

c. Sales of Services(s) 

d. Other Sales 

(e.g., rights to technology, etc) 

e. No Sales to Date 


► Skip to Question 11 
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approximate amount of total sales resulting from the technology developed during this project? 
If multiple SBIR projects contributed to the ultimate product, report only the share of total sales 
appropriate to ^ SBIR project. 

(Enter dollars[$]. If none, enter 0[zero].) 


Year when First Sale Occurred 


Total Sales Dollars of Product(s) 
Process(es) or Service(s) to Date 

Other Total Sales Dollars 

(e.g., Rights to technology, etc.) to Date 


10. To date, what percent of total sales from the technology developed during this project have 
gone to the following customers? 

(If none, enter 0[zero]. Round percentages. Answers should add to 100%) 


Private Sector 

Department of Defense (DoD) 
Prime Contractors for DoD 
NASA 

Other Federal Agencies 
State or Local Governments 
Export Markets 
Other (Specify) 

Don’t Know 


11. Expected Sales. For your company and/or your licensee, what is the approximate amount of 
total sales expected between now and the end of2002 resulting from technology developed 
during this project? (If none, enter 0 [zero]) 

If no sa les to dat e, wh at year do your expect your first sale? 

[^2001 Q 2002 I 12003 □ Later than 2003 


Total Sales Dollars of Product (s), Process(es) or Services(s) 
expected between now and the end of2002. 

Other Total Sales Dollars (e.g., rights to technology, ect.) 
expected between now and the end of 2002. 

□ Don’t Know 
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Program? 


Yes 

No ' ► Skip to Question 15 

Don’t Know 


Questions 13 & 14 apply to Fast Track Participants 

13. What sources provided matching or co-investment funding that enabled your company to 
apply for Phase II under Fast Track? 

(Check all that apply) 

a. Venture Capital Firm Provided Fimding. 

b. Another Company Provided Funding. 

c. Another Company Provided Facilities, I 

Equipment and/or Other in Kind Support. I_ 

d. Own Company Provided Funding. 

e. Don’t Know 

14. How long in months did it take to obtain and finalize Agreements(s) for third party 
funding/in kind support? 

Months 


15. Prior to your SBIR award, did your company receive funds for research or development of 
the technology in this project Jfrom any of the following sources? 

(Check all that apply) 

_Prior SBIR 

Prior non-SBIR Federal R&D 
Venture Capital 
Other Private Company 
Private Investor 


Internal Company Investment 
State or Local Government 
College or University 
Don’t Know 
No 
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16. In yolir opinion, in the absence of this SBIR award, would your company have undertaken 
this project? (Check one) 

_Definitely Not 

_Probably Not 

_Uncertain 

_Probably Yes 

_Definitely Yes 

17. What year was your company founded? 

18. Information on company founders. 

a. Number of foimders. 

b. Number of founders with a business backgroimd. 

c. Don’t Know 


19. Most recent employment of founders prior to formding this company? 
(Check all that apply) 


Other Private Company 
College or University 


Government 
Don’t Know 


20. Total revenue for the company during your fiscal year (or calendar year) 2000. 
(Check One) 

Less than $100,000 

_$100,000 to $499,999 

_$500,000 to $999,000 

_$1,000,000 to $4,999,999 

_$5,000,000 to $19,999,999 

_More than $20,000,000 
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award? 


Prior Phase I awards 
Prior Phase II awards 
Don’t Know 


22. Please give the number of patents, copyrights, and/or scientific publications for the 
technology developed during this project. 

(Enter number. If none, enter 0 [zero]) 

Number Applied for/Submitted Number Received/Published 

Patents(s) _____ _____ 

Scientific Publications |_| _ 

Don’t Know 


23. Which, if any, of the following has your company experienced as a result of the technology 
developed during this project? 

(Check all that apply) 


- Made an initial public stock offering in 
Planned an initial public stock offering. 
Established one or more spin-off companies. 
None of the above. 

Don’t Know 


(enter year) 


24. Employee Information. 

(Enter the number of employees) 

Number of employees when Phase II began 

Current number of employees 

_Number of employees hired as a direct result of the technology developed 

during this Phase II project. 










a. I believe outside investment by a third party is proof of the commercial potential of a 
research project. 

_Strongly Agree 

_Agree 

_Neither agree nor disagree 

_Disagree 

_I Strongly Disagree 

b. I would change a superior technical research approach to a slightly lesser research 
approach if I could gain access to outside investment. 

_Strongly Agree 

_Agree 

_Neither agree nor disagree 

_Disagree 

_J Strongly Disagree 

c. Rais ing the interest of outside investors for SBIR research topics is challenging. 

Strongly Agree 
_Agree 

_Neither agree nor disagree 

_Disagree 

_I Strongly Disagree 

26. Please provide the following information about the person who completed this 
questionnaire. 

Name: 

Title: 

Telephone: 

E-Mail: 

Company URL: 

27. Would you like an e-mail version of the final report? 

Yes 


No 







JLF 

Army SBIR Award Survey 

Title: 

Principle Investigator: 

Phase 11 Contract #: 

1. Is the information listed above correct? 

Yes. Go to Question 2 

No. Provide correct information below then proceed to Question 2. 
Title: 

Principle Investigator: 

Phase II Contract Number: 

2. How do you expect to commercialize yotu SBIR award? (Check one) 

_No Commercial Product, Process, or Service is Planned 

_Software 

_Intermediate Hardware Product or Component 

_Final Hardware Product 

_Process Technology 

_New or Improved Service Capability 

_Don’t Know 


Question 3 concerns whether the project is still active or if and when it was dropped. 

3, When did your company drop the project? (Check one) 

During or at the end of Phase II 
Within one year after completing Phase II. 

More than one year after completing Phase II. 


Still Active, not dropped. 
Don’t Know 
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If the technology developed during this project has led to no additional developmental 
funding or sales (and neither of these is expected to occur), skip to Question 16. 


Additional Developmental Funding and Sales 

PLEASE READ THE FOLLOWING DEFINITIONS 

Additional Developmental Funds: Include funds from federal or private sector sources, or 
from your own company, used for further development of the technology develop during the 
Phase II project. 

Sales: All sales of a product, process, or service, to Federal or Private Sector Customers 
resulting from the technology developed during this Phase II project. 

(Includes the sale of technology or rights, etc.) 


4. To date, what has been the total additional developmental funding for the technology 
developed during this project? 

(Do not include related SBIR funds received from DoD or other federal agencies. Enter dollars 
provided by each of the following sources. If none, enter zero.) 


Sources 

a. Non-SBIR Federal Funds 

b. Your Own Company 

c. Other Private Company 

d. U.S. Venture Capital Institution 

e. Foreign Venture Capital Institution 

f. Private Investor 

g. Personal Funds 

h. State or Local Governments 

i. College or Universities 

j. Other Sources (Specify) 


Dollars 



Don’t Know 
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your company’s activities with other companies and investors? 

(Check all that apply) 


Ongoing Finalized 

Negotiations Agreements 

Activities I-. I-, 

a. Licensing Agreement 

b. Sale of Complete Ownership _ _ 

c. Sale of Partial Ownership _ _ 

d. Sale of Technology Rights _ _ 

e. Joint Venture Agreement _ _ 

f. Marketing/Distribution Agreement _ _ 

g. Manufacturing Agreement _ _ 

h. Other _ _ 

i. Don’t know I_I I_ 


6. Which of the following, if any, describes the t 5 ^e and status of marketing activities by your 
company and/ or your licensee for this project? 

(Check only one block for each activity) 

Not Don’t 

Marketing Activity Planned Underway Completed Needed Know 


a. Preparation of Marketing 

b. Hiring of Marketing Staff 

c. Publicity/Advertising 

d. Test Marketing 

e. Other (Specify): 



7. Has your company and /or licensee had any actual sales of products, processes, services or 
other sales from Ae technology developed during this project? (Check all that apply.) 


a. Sales of Product(s) 

b. Sales of Process(es) 

c. Sales of Services(s) 

d. Other Sales 

(e.g., rights to technology, etc) 

e. No Sales to Date 


► Skip to Question 10 
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approximate amount of total sales resulting from the technology developed during this project? 
If multiple SBIR projects contributed to the ultimate product, report only the share of total sales 
appropriate to dus SBIR project. 

(Enter dollars[$]. If none, enter 0[zero].) 


Year when First Sale Occurred 


Total Sales Dollars of Product(s) 
Process(es) or Service(s) to Date 

Other Total Sales Dollars 

(e.g.. Rights to technology, etc.) to Date 



9. To date, what percent of total sales from the technology developed during this project have 
gone to the following customers? 

(If none, enter 0[zero]. Round percentages. Answers should add to 100%) 

Private Sector 

Department of Defense (DoD) 

Prime Contractors for DoD 

_ NASA 

Other Federal Agencies 
State or Local Governments 

_ Export Markets 

_ Other (Specify) 

Don’t Know 


10. Prior to yom SBIR award, did your company receive funds for research or development of 
the technology in this project from any of the following sources? 

(Check all that apply) 


Prior SBIR 

Prior non-SBIR Federal R&D 
Venture Capital 
Other Private Company 
Private Investor 


Internal Company Investment 
State or Local Government 
College or University 
Don’t Know 
No 
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11. In your opinionj in the absence of this SBIR award, would yoiu- company have undertaken 
this project? (Check one) 

Definitely Not 
Probably Not 
Uncertain 

_Probably Yes 

_Definitely Yes 

12. Total revenue for the company during your fiscal year (or calendar year) 2000. 

(Check One) 

Less than $100,000 

_$100,000 to $499,999 

_$500,000 to $999,000 

_$1,000,000 to $4,999,999 

_$5,000,000 to $19,999,999 

More than $20,000,000 

13. How many federal agency SBIR awards has your company received prior to this Phase II 
award? 

_Prior Phase I awards 

_Prior Phase II awards 

_I Don’t Know 

14. Please give the number of patents, copyrights, and/or scientific publications for the 
technology developed during this project. 

(Enter number. If none, enter 0 [zero]) 

Number Applied for/Submitted Number Received/Published 

Patents(s) L__J L____ 

Scientific Publications _| _ 

Don’t Know 
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(Enter the number of employees) 

_Number of employees when Phase II began 

_I Current number of employees 

_I Number of employees hired as a direct result of the technology developed 

during this Phase II project. 


16. Please provide the following information about the person who completed this 
questionnaire. 

Name: 

Title: 

Telephone: 

E-Mail: 

Company URL: 
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current SBIR S'™ '» wiren did you 

Project Status commercialize your project? Non-SBIR Additional Developmental Funding 
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